Differences in Semantic Relatedness as Judged by Humans and Algorithms

Quantifying the semantic relatedness of two terms is a field with vast amounts of research, as the knowledge provided by it has applications for many Natural Language Processing problems. While many algorithmic measures have been proposed, it is often hard to say if one measure outperforms another, since their evaluation often lacks meaningful comparisons to human judgement on semantic relatedness. In this paper we present a study using the BLESS data set to compare the preferences of humans regarding semantic relatedness to popular algorithmic baselines, PMI and NGD, which shows that significant differences in relationship-type preferences between humans and algorithms exist.